Method and apparatus for exemplary segment classification

ABSTRACT

Method and apparatus for segmenting speech by detecting the pauses between the words and/or phrases, and to determine whether a particular time interval contains speech or non-speech, such as a pause.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a Continuation Application of U.S.application Ser. No. 14/262,668, filed Apr. 25, 2014, which claims thebenefit of U.S. Provisional Patent Application No. 61/825,523, filed onApr. 25, 2013, in the U.S. Patent and Trademark Office, the disclosureof which is incorporated herein by reference in its entirety.

BACKGROUND

1. Field

Speech segmentation is the process of identifying the boundaries betweenwords, syllables, or phonemes in spoken natural language. In all naturallanguages, the meaning of a complex spoken sentence (which often hasnever been heard or uttered before) can be understood only bydecomposing it into smaller lexical segments (roughly, the words of thelanguage), associating a meaning to each segment, and then combiningthose meanings according to the grammar rules of the language. Therecognition of each lexical segment in turn requires its decompositioninto a sequence of discrete phonetic segments and mapping each segmentto one element of a finite set of elementary sounds (roughly, thephonemes of the language).

For most spoken languages, the boundaries between lexical units aresurprisingly difficult to identify. One might expect that the inter-wordspaces used by many written languages, like English or Spanish, wouldcorrespond to pauses in their spoken version; but that is true only invery slow speech, when the speaker deliberately inserts those pauses. Innormal speech, one typically finds many consecutive words being saidwith no pauses between them.

2. Description of Related Art Voice activity detection (VAD), also knownas speech activity detection or speech detection, is a technique used inspeech processing in which the presence or absence of human speech isdetected. [1] The main uses of VAD are in speech coding and speechrecognition. It can facilitate speech processing, and can also be usedto deactivate some processes during non-speech section of an audiosession: it can avoid unnecessary coding/transmission of silence packetsin Voice over Internet Protocol applications, saving on computation andon network bandwidth.

SUMMARY

Aspects of the exemplary embodiments relate to systems and methodsdesigned to segment speech by detecting the pauses between the wordsand/or phrases, i.e. to determine whether a particular time intervalcontains speech or non-speech, e.g. a pause.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of a computer system for segmentingan input audio wave into speech segments, according to an exemplaryembodiment.

FIG. 1a illustrates another block diagram of a computer system forsegmenting an audio waver into segments, according to an exemplaryembodiment.

FIG. 1b illustrates another block diagram of a computer system forsegmenting an audio waver into segments, according to an exemplaryembodiment.

FIG. 1c illustrates another block diagram of a computer system forsegmenting an audio waver into segments, according to an exemplaryembodiment.

FIG. 2 illustrates a flow diagram of a method of detecting pauses inspeech, according to an exemplary embodiment.

FIG. 3a illustrates a graphical representation the energy of an inputaudio as a function of time.

FIG. 3b illustrates a graphical representation of the variance of theenergy of the input audio within a given window as a function of time.

FIG. 4a illustrates a graphical representation of the location of pauseswithin the input audio wave

FIG. 4b illustrates a graphic representation of kurtosis identifying thebest place to divide a speech segment.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

FIG. 1 illustrates a block diagram of a system for detecting pauses inspeech, according to an exemplary embodiment.

The pause detecting system in FIG. 1 may be implemented as a computersystem 110 is a computer comprising several modules, i.e. computercomponents embodied as either software modules, hardware modules, or acombination of software and hardware modules, whether separate orintegrated, working together to form an exemplary computer system. Thecomputer components may be implemented as a Field Programmable GateArray (FPGA) or Application Specific Integrated Circuit (ASIC), whichperforms certain tasks. A unit or module may advantageously beconfigured to reside on the addressable storage medium and configured toexecute on one or more processors or microprocessors. Thus, a unit ormodule may include, by way of example, components, such as softwarecomponents, object-oriented software components, class components andtask components, processes, functions, attributes, procedures,subroutines, segments of program code, drivers, firmware, microcode,circuitry, data, databases, data structures, tables, arrays, andvariables. The functionality provided for in the components and unitsmay be combined into fewer components and units or modules or furtherseparated into additional components and units or modules.

Input 120 is a module configured to receive human speech from any audiosource, and output the received speech to the energy calculator 130. Theaudio source may be live speech, for example received from a microphone,recorded speech, for example speech recorded in a file, synthesizedspeech, etc. Energy Calculator 130 is a module configured to receive thespeech output by the input module 120, calculate the energy of thewaveform of the human speech, and output the calculated energy of thewaveform to the calculator 140. The calculator 140 is a moduleconfigured to calculate the variance of the energy of the speech, basedon the energy of the waveform output by the energy calculator 120, andoutput the calculated variance to the segmenter 150. The segmenter 150is configured to receive the variance calculated by the calculator 140,break the speech into segments based upon the audio characteristic ofthe speech, and output the segments to the refining segmenter 160.Refining Segmenter 160 is configured to receive the segments from thesegmenter 150 and further divide individual segments in which theindividual segment duration is greater than acceptable for the intendedapplication, e.g. an automatic speech recognizer (“ASR”) which can onlyprocess 25 seconds of speech, or a close captioning system which canonly process 10 second of speech.

FIG. 1a illustrates a component level diagram of the embodiments. Input120 is a module configured to receive human speech from any audiosource, and output the received speech to Computer system 110. Input 120may be a live speaker, a module configured to stream audio, a feed froma videoconference with audio, a module configured to stream audio andvideo and/or a module configured to download or store audio oraudio/video files. Computer System 110 is the computer system describedabove. ASR 175 is configured to receive the speech segments outputted byComputer System 110 and convert said segments into text. Text String 185is the resulting output from the ASR.

FIG. 1b illustrates a component level diagram further adding VideoSynchronizer 185 configured to receive Text String 185 and synchronizeit with any video that was received from Input 120 such that it can bereceived by Closed Captioning 185 a, a module configured to receive thesynchronized a text and video and display it as closed captioning.

FIG. 1c illustrates a component level diagram further adding Translation190, a module configured to translate the closed captioning text receiveclosed captioning 185 a, TTS 190 a, a module configured as a text tospeech engine to convert the translated text into speech, and AudioTrack 190 b, a module configured to synchronize the audio output fromTTS 190 a with the input video to create an translated audio track.

FIG. 2 illustrates a flow diagram of a method detecting pauses inspeech, according to an exemplary embodiment.

At step 210, Input 120 receives input audio data from any audio sourcewhich may include live speech, streaming audio and/or audio/video,downloaded audio and/or audio/video. Said audio source could be a livespeech, recorded speech, synthesized speech, etc. At step 220 Calculator120 calculates the energy E of speech waveform Y. In one embodiment,Calculator 120 determines the energy of the Y over an n second interval(t, t+n). Specifically

E=Y(t, t+n)̂2   Eq. 1

An n second interval is a standard interval commonly used amongst energycalculators.

At step 230, Calculator 140 calculates the variance S, or standarddeviation, of the Energy E over an x second energy sample window whichslides as energy is sampled at each n internal. Calculating the varianceS is within the knowledge of one skilled in the art.

At step 240, Segmenter 150 segments Input 120 into speech segments witha granularity of n second interval. Segmenter 150 uses S to determinethe beginning and ending of each speech segment in the waveform as willbe discussed in detail below. At step 250, Segmenter 150 compares S withan upper and lower threshold determined according to exhaustive testing.In one embodiment of computer system 101, the upper and lower thresholdsare one standard deviation from the mean of S over the entire speech. Atstep 260 a, when the variance is greater than the upper threshold,Segmenter 150 classifies the interval as speech. Speech tends to have Sdue to its rhythmic nature. When the variance is below the lowerthreshold at step 260 b Segmenter 150 determines the interval asnon-speech. Non-speech generally lacks the high energy variance rhythmicnature of speech/When the variance is between 2 the upper and lowerthreshold, at step 260 c, Segmenter 150 determines the interval to bethe same as the speech or non-speech determination of the previousinterval. This gives stability to the classification, i.e. if thevariance hovers at either the upper or lower threshold, segmenter 150won't repeatedly change classification over a small time interval.

The segmenter 150 may classify segments as speech or non-speech asdescribed above, for several reasons which include, reduce the amount ofinformation that needs to be sent to an ASR, i.e. only speech is sent toan ASR.

At step 270, Refining Segmenter 160 selects segments which are too longin length by time, for the requirements of the desired output device,e.g. ASR, close captioning display, etc. For example, some speechrecognizers can only accept segments of a given time length while closecaptioning standards for television limits the display of closedcaptioned information to ten seconds of speech. The determination oflarge segment is user determined based upon this desired end result. Atstep 280, Refining Segmenter 160 determines the maximum kurtosis of themiddle region of a segment which is too large and divides the segmentsinto smaller segments at the point of lowest kurtosis. Kurtosis is anymeasure of the “peakedness” of the probability distribution of areal-valued random variable, in this case, the input speech. At step 280this division is repeated until all segments are of shorter timeinterval that the user set threshold.

Experimentally, dividing the large segment at the maximum kurtosis ofthe middle third is optimal. The goal is to divide the large segment atthe best place for a speech recognizer to recognize the audio inputconsidering such items as time constraints, time lag, etc., e.g. changeof tone, pauses too short to be detected by changes in variance, anddoes not result in oversegmentation, e.g. dividing words or phrases.

FIG. 3a is a graph of the Energy E of a sample audio wave of the phrase“Um, what was it?” followed by background noise. Time Interval 305represents the time interval of the word “Um”; the sampled intervalsover which the word is spoken in the input speech). Energy 305 aillustrates the energy of the word “Um” over Time Interval 305. TimeInterval 310 represents the time interval over which there is no speech,i.e., the natural pause between “Um” and “what was it”. Consequently itsenergy, Energy 310 a is low and relatively unvarying, i.e. S is lessthan Threshold 345 as detailed in FIG. 3 b. Time Interval 315 representsthe time interval spanning the remainder of the phrase, i.e. “what wasit”. Time Interval 315 has insignificant pauses which are not detectedby the classifying the variance in the energy. Energy 315 a representsthe energy of the speech over Time Interval 315. Time Interval 320represents the background or motor noise. Energy 320 a represents theenergy of the background noise. Observationally, background noise tendsto have relatively constant energy as shown in Energy 320 a, while humanspeech is rhythmic, which has high variance.

FIG. 3b illustrates the variance of the energy E of the spoken phraseover sliding window discussed above. The variance of the energy is usedto determine whether a particular time interval contains speech ornon-speech. For purposes of this disclosure, non-speech or pausesincludes, background noise, silence, etc. The input audio is identifiedas speech when the variance over the sliding window is greater thanThreshold 340, signifying the rapid rhythmic changes characteristic ofhuman speech and identified as non-speech when the variance is less thanThreshold 345 signifying the gradual changes inherent is non-speech suchas silence, motor noise, etc. When the audio is simultaneously greaterthan Threshold 345 and less than Threshold 340, its identification, i.e.speech or non-speech, remains unchanged from the previous identificationuntil the variance crosses either Threshold 340 or Threshold 345. Asexplained above, this provides stability to the classifications.Experimentally Threshold 340 is one standard deviation below the mean ofS across the entire input speech, while Threshold 345 is one standarddeviation above the mean of S across the entire input speech.

In one embodiment of the invention, the segmentor 150 lacks the entireinput speech, e.g. when Input 120 is a live speaker, streaming audio,etc. The mean of S is continually recalculated as more speech isobtained.

Time index 350 illustrates the variance at the beginning of the phrase“Um, what is it?” Time index 351 illustrates the first point where thevariance exceeds Threshold 340 and is identified by Computer System 110as speech, i.e. sample at point of large variance change begins speech?When Input 120 is a live speaker, streaming audio, or any input wherethe entire speech and its duration is unknown, computer system 110stores the segment marked as speech in memory. Where input 120 is adownloaded file, or any input where the entire speech and its durationare known, computer system 110 stores only the time indexes of thebeginning and ending time of each speech segment. Computer System 110identified the audio input as speech until the variance becomes lessthan Threshold 345 at Time Index 352. The audio input remains identifiedas non-speech until Time Index 353, where the variance once againexceeds Threshold 340. The audio is identified as non-speech at TimeIndex 354 when the variance becomes less than Threshold 345.

FIG. 4a illustrates a representative waveform; Waveform 404 where thevarious lexical segments of the input audio “Um, where is it?” has beenidentified as speech or non-speech. Region 405, the “Um”, is identifieda speech. Region 406 is identified as non-speech. Region 430 the phrase“where is it?” is identified as speech. Region 440, the motor in thebackground is identified as non-speech.

For purposes of this disclosure, Segment 405, the portion of Waveform404 contained within Region 405, is within the acceptable size limit fora speech segment. Segment 430, the portion of Waveform 404, containedwithin Region 430, exceeds the acceptable size limit for a speechsegment so it must be broken into smaller segments. The acceptable sizelimit for a segment is user determined. A speech segment has a durationthat it too large if it causes acceptable lags, exceeds the threshold ofthe output device, e.g. ASR, etc. A speech segment is too small if itcontains too little information, i.e. an incomplete phrase, such thatthe ASR lack context when attempting to recognize a word.

FIG. 4b illustrates the Kurtosis of the waveform 404 at Segment 430.Computer System 101 uses the Kurtosis as described below to determinethe best place as described above to divide the segment. Computer System101 divides Waveform 405 into Region 410, Region 415, and Region 417,calculates the Kurtosis over a Region 415 and selects Time Index 435,the local Kurtosis maximum, as the division point for Waveform 404.Computer system 110 chooses a middle region to avoid dividing segmentsnear the speech segment edge where kurtosis tends to be higher thusgenerating many excessively small speech segments. In one embodiment ofthe invention, Region 410, Region 415, and Region 417 are each one thirdof the segment.

Computer system 110 repeats the process of dividing segments until eachsegment, is within the acceptable size limit, including segments createdfrom the division of larger segments.

What is claimed is:
 1. A system configured to identify lexical segmentsin speech, the computer system comprising: an input interface configuredto receive an input of audio; an energy calculation module configured tocalculate energy of the audio over first time intervals of the audio; avariance calculation module configured to calculate variance of theenergy in the first time intervals of the audio over every second timeinterval within the first time intervals that slides as the energycalculation module calculates the energy of the audio at each of thefirst time intervals; a speech segmentation module configured determinefor the audio over each of the second time intervals that the audio overeach of the second time intervals contains speech based on the varianceof the energy, and determine lexical segment boundaries of the speechwithin the audio based on each of the second time intervals of the audiothat contains speech; and a speech segmentation refinement moduleconfigured to segment the speech within the lexical segment boundariesand output the segmented speech to an output device configured to outputthe speech according to kurtosis of the speech within the lexicalsegment boundaries.
 2. The system of claim 1, wherein the kurtosis ofthe speech is a measure of peakedness of a probability distribution ofthe speech.
 3. The system of claim 1, wherein the speech segmentationmodule is further configured to determine whether the variance of theenergy of the audio over each second time interval is greater than athreshold value of variance of the energy of the audio, determine forthe audio over each of the second time intervals that the audio overeach of the second time intervals contains speech in response todetermining that the variance of the energy of the audio over eachsecond time interval is greater than the threshold value, and determinelexical segment boundaries of the speech within the audio based on eachof the second time intervals of the audio that contains speech.
 4. Thesystem of claim 3, wherein the speech segmentation module is furtherconfigured to determine whether the variance of the energy of the audioover each second time interval is less than a lower threshold value ofvariance of the energy of the audio one standard deviation below a meanvariance of energy of the audio signal, and determine for the audio overeach of the second time intervals that the audio over each of the secondtime intervals does not contain speech in response to determining thatthe variance of the energy of the audio over each second time intervalis less than the lower threshold value.
 5. The system of claim 4,wherein the speech segmentation module is further configured todetermine whether the variance of the energy of the audio over eachsecond time interval is greater than the lower threshold value ofvariance of the energy of the audio and less than the threshold value ofvariance of the energy of the audio, and determine for the audio overeach of the second time intervals having variance of the energy greaterthan the lower threshold value and less than the threshold value ofvariance that the audio over each of the second time intervals containsspeech if a previous segment of the audio over a previous second timeinterval is determined to contain speech and does not contain speech ifthe previous segment of the audio over the previous second time intervalis determined to not contain speech.
 6. The system of claim 1, whereinthe speech segmentation refinement module is further configured tosegment the speech within the lexical segment boundaries at a maximum ofthe kurtosis.
 7. The system of claim 6, wherein the speech segmentationrefinement module is further configured to segment the speech within thelexical segment boundaries in thirds, determine the maximum of thekurtosis in a middle third of the speech, and segment the speech intothe segmented speech at the maximum of the kurtosis in the middle thirdof the speech.
 8. The system of claim 7, wherein the speech segmentationrefinement module is further configured to recursively segment thesegmented speech while time durations of the segments of the segmentedspeech are greater than a threshold.
 9. The system of claim 8, whereinthe threshold is a length of time that the output device is able toprocess.
 10. The system of claim 9, wherein the output device comprisesone of an automatic speech recognizer (ASR) and a closed captioningsystem.